MagicDraw VIATRA benchmark results
The goal of this benchmark is to measure the query speed and memory overhead of the VIATRA query framework using organic SysML model instances. The measurements are taking place in a MagicDraw-based ecosystem. During the benchmark a properties of various queries are measured, while they are being executed using a number of query engine implementations.
Measured metrics:
-
Initialization time: The time it takes to initialize the VIATRA query framework on a loaded model.
-
Query result calculation time: The time it takes to fetch the result set of a given query over an already initialized VIATRA query engine.
-
Query result set size: Size of the result set of a selected query over the given model instance.
-
Overall memory usage: Overall allocated memory of the measured MagicDraw instance after querying.
-
Memory overhead of VIATRA: Memory overhead added by initializing a VIATRA query engine over a model instance, and executing a given query on it.
During the execution of the benchmark the following environment was used (as a Jenkins slave on build.incquerylabs.com):
-
MagicDraw 18.5
-
Installed plugins:
-
Runtime JVM memory parameters
-
Heap size: -Xmx8G -Xms4G
-
Stack size: -Xss1024K
-
-
Server specifications
-
Memory: 16GB DDR3 @ 1600 MHz
-
Disk: 120 GB Samsung SSD 850 EVO
The benchmark is executed on various sized instances of the Thirty-meter-telescope (TMT) model. These instances are created via copy-pasting the work packages of the original TMT model. Each added copy is on a new structure level, resulting in deep model hierarchy. This way the size of the model does not hinder the MagicDraw UI unnecessarily.
Sizes of each used model instance (number of total EObjects)
-
300000 Elements (Default TMT)
-
540000 Elements (TMT 2X)
-
780000 Elements (TMT 3X)
-
1040000 Elements (TMT 4X)
-
1260000 Elements (TMT 5X)
The VIATRA Query Framework features multiple query evaluation backend implementations.
-
Incremental backend based on Rete networks: Stores individual constraints of queries along with their result sets in a graph network (Rete network). This network serves as a cache that is maintained on the fly as the model instance is modified. More information about the Rete algorithm can be found in the following article.
|
Note
|
Queries in the InstaSchema MagicDraw plug-in utilize the Rete backend |
-
Local search backend: With the local-search backend, the search is restarted from scratch each time a query evaluation is requested. The search is executed based on a search plan that selects the first node to be matched and defines the order in which individual query constraints are evaluated. More information about the Rete and LS algorithms can be found here
-
Hybrid backend: Combines the positive aspects of Rete and LS based querying. Allow the user to mark conditions to be executed solely by the Rete engine, while others will be executed using local search. This way certain patterns can be optimized to be run with the backend that match their characteristics the best.
Returns classes with 'Block' or 'Requirement' SysML stereotype, or classes with 'Constraint' SysML stereotype that are not involved as part of a dependency
/**
* Disjunction reusing the blocksOrRequirements and constraintBlocks patterns and the negation of dependencies.
*/
pattern blocksOrRequirementsOrConstraints(class : Class) {
find blocksOrRequirements(class);
} or {
find constraintBlocks(class);
neg find dependencies(class, _); /
neg find dependencies(_, class);
}
/**
* Disjunction of blocks and requirements.
*/
pattern blocksOrRequirements(class : Class) {
find blocks(class);
} or {
find requirements(class);
}
/**
* Querying all instances of an EClass that have an instance of the Stereotype 'Block'
*/
pattern blocks(class : Class) {
Class.appliedStereotypeInstance.classifier.name(class, "Block");
}
/**
* Querying all instances of an EClass that have an instance of the Stereotype 'Requirement'
*/
pattern requirements(class : Class) {
Class.appliedStereotypeInstance.classifier.name(class, "Requirement");
}
/**
* Querying all instances of an EClass that have an instance of the Stereotype 'Constraint'
*/
pattern constraintBlocks(class : Class) {
Class.appliedStereotypeInstance.classifier.name(class, "Constraint");
}
/**
* Checks if there is a dependency relation between two Named Elements
*/
pattern dependencies(source : NamedElement, target : NamedElement) {
Dependency.supplier(dependency, target);
Dependency.client(dependency, source);
}Queries states with more than 1 incoming transition and their transitive substates as well. (Here transitive substate means every state that is in a region contained by the 'state' parameter in a way)
pattern transitiveSubstatesWithCheck3(state : State, transitiveSubstate : State) {
find statesWithMoreIncomingTransitions(state);
//Transitive closure
find parentState+(transitiveSubstate, state);
}
pattern statesWithMoreIncomingTransitions(state : State) {
transitionCount == count find incomingTransitions(_, state);
check (transitionCount > 1);
}
/**
* Simple pattern for transitive closures.
* Checks parent-child relation between states
*/
pattern parentState(state : State, parentState : State) {
State.region(parentState, subregion);
Region.subvertex(subregion, state);
}Queries NamedElement pairs which are connected by a dependency and the source elements' name is shorter than the target’s.
pattern alphabeticalDependencies(source : NamedElement, target : NamedElement) {
find namesOfDependencyEndpoints(source, sourceName, target, targetName);
check (sourceName < targetName);
}
pattern namesOfDependencyEndpoints(source : NamedElement, sourceName : java ^java.lang.String, target : NamedElement, targetName : java ^java.lang.String) {
find dependencies(source, target);
NamedElement.name(source, sourceName);
NamedElement.name(target, targetName);
}Queries Dependency elements that are part of a dependency circle. Allowing fast circular dependency detection.
/*
* Pattern responsible for detecting circular dependency chains.
*/
pattern circularDependencies(dependency : Dependency) {
find dependencyChains+(dependency, dependency);
}
/**
* Pattern describing succession relation between two dependencies.
*/
pattern dependencyChains(source : Dependency, target : Dependency) {
Dependency.supplier(source, elem);
Dependency.client(target, elem);
}Queries the following tuple:
-
State
-
Transition
-
Trigger
-
Event
-
Activity
Where the state has a loop transition that has a trigger, some effect as an Activity, has no guard and defines an event.
/**
* A loop transition that
* - Has a trigger
* - Has some effect that is an activity
* - Has no guard
* - defines an event
*
*/
pattern loopTransitionWithTriggerEffectEventNoGuard(state : State, transition : Transition, trigger : Trigger, event : Event, effect : Activity) {
Transition.source(transition, state);
Transition.target(transition, state);
Transition.kind(transition, ::external);
Transition.trigger(transition, trigger);
Transition.effect(transition, effect);
neg find transitionWithGuard(transition, _);
Trigger.event(trigger, event);
}
private pattern transitionWithGuard(trans : Transition, guard : Constraint){
Transition.guard(trans, guard);
}Queries transitions that connect two states with the following properties:
-
Source state is a member of a composite state
-
Target state is a member of the same state machine, however not contained in a composite state
-
Target state has an outgoing transition towards a final state
pattern transitionPointingOutOfCompState(source : State, target : Vertex, transition : Transition) {
Transition.source(transition, source);
Transition.target(transition, target);
//pattern for target side element
Vertex.container(target, targetRegion);
Region.stateMachine(targetRegion, _);
Vertex.outgoing(target, targetVertexTransition);
Transition.target(targetVertexTransition, finalState);
FinalState(finalState);
//Pattern for source side element
find parentState+(source, sourceParent);
State.container.stateMachine(sourceParent, _ );
}
/**
* Simple pattern for transitive closures.
*/
pattern parentState(state : State, parentState : State) {
State.region(parentState, subregion);
Region.subvertex(subregion, state);
}Queries the state with the largest amount of transitive substates whose names are shorter than 6 characters.
pattern stateWithMostSubstates(state : State) {
// the number of substates of this state
find numberOfTransitiveSubstates(state, count);
// is the maximum from all states
count == max find numberOfTransitiveSubstates(_, #substateCount);
}
pattern numberOfTransitiveSubstates(state : State, transitiveSubstateCount : java ^java.lang.Integer) {
transitiveSubstateCount == count find transitiveSubstatesWithCheck(state, _);
}
pattern transitiveSubstatesWithCheck(state : State, transitiveSubstate : State) {
find statesWithShortNames(state);
find parentState+(transitiveSubstate, state);
}
/**
* Returns states with short names.
*/
pattern statesWithShortNames(state : State) {
State.name(state, name);
check(name.length < 6);
}Reuses every other benchmarked query in an 'OR' relation. This way overall memory overhead can be measured if a number queries are used.
pattern allBenchmarkedQueries(param : NamedElement) {
find blocksOrRequirementsOrConstraints(param);
} or {
find transitiveSubstatesWithCheck3(param, _);
} or {
find alphabeticalDependencies(param, _);
} or {
find circularDependencies(param);
} or {
find loopTransitionWithTriggerEffectEventNoGuard(param, _, _, _, _);
} or {
find transitionPointingOutOfCompState(param, _, _);
}-
Benchmark job URL: https://build.incquerylabs.com/jenkins/job/MagicDraw-Tools-Benchmark/66/
-
Benchmark result archive: https://github.com/IncQueryLabs/magicdraw-viatra-benchmark/releases/tag/Run66
Note that in this case there are more data sets than in the other cases. This is down to the fact, that apart from the standard Rete and LS based execution, this query is also measured via an additional query evaluation method.
-
Hybrid: In case of the hybrid execution, the patterns calculating the transitive closure are evaluated using the Rete-based backend, while the rest of the pattern is evaluated using local search.
